Users have long used a keyboard, mouse, trackpad, or touch screen to control an electronic device with a display screen, e.g., desktop PC, netbook, tablet, smart phone, etc. However it can be more natural for the user to control such a device using gestures, perhaps hand movements made in proximity to the display associated with the device. Such gestures require little practice compared to mastering a keyboard, and do not require the user to look away from what might be displayed by the device. Further there is less repetitive motion, which can decrease likelihood of repetitive stress injury (RSI) to the user.
Systems and methods to detect and recognize gestures, including three-dimensional gestures, can vary from the highly complex and expensive to the relatively straightforward and inexpensive. However implemented, such systems and methods seek to enable a user to control an electronic device including controlling what appears on the device display screen, preferably without necessarily touching the electronic device. The display screen may but is not required to be an integral portion of the device housing. While user touching a touch screen device is a common mode of scrolling text, selecting menu items, manipulating graphics and the like, there are several disadvantages to user interactions that rely upon a touch screen. Touching tends to smudge the device touch screen with fingerprints, and requires the user to literally be within an arm's length of the touch screen surface. Touch screens simply are not feasible for a large sized device, perhaps a smart TV, that may be wall mounted. This problem is aggravated when user view streaming video and the like over such large screen devices in that they tend to sit some distance away from the screen to enjoy the presentation. Even if they could do so, the user would have to stand up and approach the smart TV to interact by touching its display screen. Of course a user could employ a remote control but that would require holding another device. Portable devices such as tablets may have touch screens and often have flimsy stands to prop-up the device for ease of user touching. But it is too easy for a user to accidently tip such devices over when interacting by touching the screen. Handheld devices such as smart phones have relatively tiny display screens, and requiring a user to touch the screen surface physically restricts the area of user interaction to a relatively small region. Just as repetitive keyboard typing is known to cause stress injury, it is recognized that making certain types of repetitive gestures can cause injury known as ‘gorilla arm syndrome’. (See http://www.wired.com/gadgetlab/2010/10/gorilla-arm-multitouch/.) Some devices provide for user interaction within a zone slightly spaced-apart from the display screen surface, without touching the surface. But the user is required to hold his/her hand very close to the screen surface within the screen boundaries. Such movements are not feasible for a wall mounted device, and can be difficult to accomplish for small portable devices such as smart phones.
A graphical user interface (GUI) is a favored mode of user-device interaction and refers to user selection of objects presented on the device display screen. Examples of selectable GUI objects may include icons, buttons, check boxes, menus and menu items, spreadsheet cells, location in a GUI canvas such as a location of a displayed map, a weapon trigger in a displayed video game, etc. The term selection refers to the act of selecting a selectable GUI object. The term “detection” refers to the software code that detects and maps the user object to a predefined coordinate system. Detection further identifies the target item with which the user intends to interact. The user's intent, what the user wants to do with the selected object, may be defined as part of the selection or by the next user action. For instance, user selection of a displayed menu item may open up a list of sub-menus, or a so-called ribbon. It is known to enable the user to select an edge or corner of the display whereat no object need be displayed to reveal hidden menus that can provide additional user options. If the displayed object is perhaps a geographic map, user selection of a map location can signal to the device and its detection system a ‘grab’ function. Subsequent user created motion of the map selected point can scroll the map in the direction of the motion.
Gestures are a form of human communication using body parts, e.g., user fingers, hands, face, head, shoulders, torso, feet, toes, etc. Gestures are called natural interfaces because they can be performed without use of any additional item, e.g., no stylus, no special gloves, etc. need be used. Gestures can also input data to computers and thus augment other forms of natural interfaces such as speech.
User gestures occur in a real world situation, and their recognition requires a reliable method of three-dimensional input so they can be detected and parsed for identification using computer software. When a gesture is used to communicate with a computer, higher vocabulary of gesture types or gesture language may be defined. For instance, a clockwise or counter-clockwise user hand rotation gesture may be used to mimic increasing or decreasing volume of a virtual rotary knob perhaps displayed on the device screen.
How a gesture is interpreted can be customized to various applications on the computer device. For instance, a waving type gesture can be used to get the attention of the computer device to identify a user about to communicate with the computer device. The computer device consequently can enter a mode to accept subsequent user gestures. A gesture may be defined to start a specific application, or even to unlock or lock the computer device.
Certain gestures are known in the prior art, for example “pinching” or “expansion” and “pausing”, as are use of gestures for three-dimensional guided navigation, as in motioning left or right with a user finger. TABLE 1 below represents a survey undertaken by applicant of prior art gestures. In Table 1, ten digit references are pending U.S. patent applications. Issued U.S. patent numbers are prefaced with the abbreviation USP. Applicant's recital of the below references is not to be construed as an admission by applicant that the references should indeed be patentable or are necessarily relevant to aspects of the present invention described here. The references are merely cited as a broad survey of what the prior art generally seems to encompass relative to gestures.
TABLE 1TaxonomyReferenceDisclosure3Dgestures—curtain2012/0274550 “”Gesture Mapping for3D hand gesture moves laterally to cause application toDisplay Device” (Campbell, 2010)turn reading material page [0022, FIG. 105A]; movingentire hand vertically is similar to vertical curtain [0026,FIG. 106B]3D gestures—pinch2012/0274550 “Gesture Mapping for3D pinching using thumb and forefinger pinched together,and zoomDisplay Device” (Campbell, 2010)pinching and then rotating hands to rotate object [0027]3D gestures—pinch & U.S. Pat. No. 8,086,971 “Apparatus, Methodsacoustic sensing—zoom in/zoom out by expanding/zoomand Computer Program Productsclosing two fingers including the thumb (FIGS. 5A, 5B, 8D,Providing Finger-Based and Hand-p. 6 line 20)Based Gesture Command” (Nokia,2006) . . .3D gestures—pinch2012/0204133 “Gesture-Based Userperforming grab gesture by pinching thumb with two orInterface” (Primesense 2012)more fingers [0061, FIG. 6B]3D gestures—pausing U.S. Pat. No. 7,665,041 “Architecture forpausing full hand movement to interact with the systemControlling a Computer Using Handand perform tasks such as select, bring window toGestures” (Microsoft), 2003)foreground [p17, lines 55-65]3D gestures—pausing 2009/0103780 “Hand-Gesturepoking and pausing with extended index finger [0048, FIG.Recognition Method” (Nishihara),5]2008)3D gestures—pausing U.S. Pat. No. 8,086,971 “Apparatus, Methods3D pausing—finger stays motionless to select an objectand Computer Program Productsdisplayed on the screen (p. 9, line 3).Providing Finger-Based and Hand-Based Gesture Command” (Nokia,2006)3D gestures—2011/0219340 “System and Methodpointing with index finger [FIG. 1 defines a “pecking handpoking/pausingfor Point, Select and Transfer Hand gesture” and is a pecking action [0027]Gesture Based User Interface”(Pathangay, 2010)3D gestures—poking8,086,971 “Apparatus, Methods Andclicking with a poking/pointing single finger or moving the(clicking)Computer Program Products Providingfinger slowly under a displayed object (p. 6, line 5), andFinger-Based and Hand-Basedsubsequent tapping by a single finger [p8, line 65]; fingerGesture Command” (Nokia, 2009))stays above an object followed by a slow movement [p9,line 5]3D gestures—cocking20090103780 “Hand-Gestureextended index finger and extended thumb, user thumb(thumb cocking)Recognition Method” (Nishihara, 2008)movement [0052, FIG. 5]. (Less cocking, more movementof thumb closer to index finger)3D gestures—cocking20090103780 “Hand-Gesturecocking with index finger [0062]; pointing with index finger(thumb)Recognition Method” (Nishihara, 2008)[Claim 31]3D gestures—others2012/0274550 “Gesture Mapping Formoving entire hand vertically, similar to a vertical curtainDisplay Device” Campbell 2010)[0026, FIG. 106B]3D gestures—others8,086,971 “Apparatus, Methods andmultiple 3D gestures—open closed hand, make circle withComputer Program Products Providingtwo fingers, clockwise/anticlockwise rotation with singleFinger-Based and Hand-Basedfinger [FIG. 4B], double click by two fingers, moving twoGesture Command” (Nokia-2006)fingers. Switching on/off based on change in direction offingers movement, or alternatively finger acceleration (p. 6lines 0-20)(p8-9)3D gestures—othersU.S. Pat. No. 8,96,114 “Gesture Recognitionvirtual dry-erase board function on which users can drawInterface System With Vertical Display” or write [p5, line 40]; user must touch display screen with(Northrop, 2008the index finger only, or with index and thumb finger, orpalm of hand to initiate the dry erase board in differentcontexts. [p. 11, lines 50-70, p. 12]3D gestures—othersU.S. Pat. No. 7,665,041 “Architecture foruses both 3D gestures and voice to interact withControlling a Computer Using Handcomputer [p17, lines 40-50]; bringing fingers together withGestures”” (Microsoft, 2003thumb and subsequently moving full hand to bring awindow to the forefront [p18 lines 5-15]; moving fingersvertically [p18, lines 20-35]; has “record gesture mode”for users to record new gestures [p 18, lines 45-55]; alsouses voice recognition to interact with gestures [Claim 24]3D gestures—others2009/0103780 “Hand-Gestureboth single hand and double hand gestures pointing bothRecognition Method” (Nishihara, 2008)index fingers [0047, 0056, FIG. 6]; flat extended hand, withfingers together or spread out [FIG. 5, 0049]; contact withthe display surface with at least one finger—index fingerand thumb, two fingers [0050, FIG. 5]; opening all fingersof closed hand [FIG. 7]; cocking with index finger [0062];well described library of gestures3D gestures—othersU.S. Pat. No. 7,340,077 “Gesture Recognitionrich hand gestures, such as directing one hand at anSystem Using Depth Perceptiveelectronic device while raising two fingers [p12, line 20]Sensors” (Gokturk, 2003))3D gestures—othersU.S.Pat. No. 8,96,114 “Gesture Recognitiontouching a vertical surface with a raised index finger andInterface System With Vertical Display” a closed fist [p 11, line 60]; touching the surface with a(Northrop, 2008)flat palm [p12, line 60]3D gestures others2012/0204133 “Gesture-Based Usergrab (closing hand to make a fist or folding one or moreInterface” (Primesense, 2012)fingers towards palm) [0060, FIG. 6A]; release (openinghand from closed or folded state to release Grab) [0062,FIG. 6C]; push/pull (moving hand towards an applicationobject or moving hand away) [0063, FIG. 7]; rotating palmclockwise/counter clockwise [0065, FIG. 8A, 8B]3D gestures — others2011/0219340 “System and Methodgrabbing or half grabbing with full hand either closed orfor Point, Select and Transfer Handopen [0042-0043]Gesture Based User Interface”(Pathangay, 2010)3D gestures—guided2010/0199228 “Gesture Keyboarding”can use 3D gestures to type onto a displayed virtual(based on menu)(Microsoft, 2009)keyboard [0093, FIG. 6]3D gestures guidedU.S. Pat. No. 7,665,041 “Architecture forhand gestures point to, expand/close menus [0061]Controlling a Computer Using HandGestures” (Microsoft), Mar. 25, 20033D gestures guidedU.S. Pat. No. 8,086,971 “Apparatus, Methodsone finger stays above an object, then an object-specificand Computer Program Productsmenu appears, the other finger performs circular rotationsProviding Finger-Based and Hand-and toggles through menu [p9, line 35]Based Gesture Command” (Nokia,2006)entering 3D Interaction U.S.Pat. No. 8,96,114 “Gesture Recognitionoption to enter into virtual dry erase white board mode byzone (waving)Interface System With Vertical Displaytouching display screen with one finger, with hand mostly(Northrop, 2008)closed; application then interprets future gestures as tobe displayed in real time on virtual dry erase board. [p11,lines 55-65]entering 3D interaction U.S. Pat. No. 7,665,041 “Architecture forcan trigger system attention by making specific gesture,zone (waving)Controlling a Computer Using Hande.g. raising hand [p 21, lines 5]; discloses rich gestureGestures”_(Microsoft, 2003)library (See FIGS. 7, 8]3D gesturesU.S. Pat. No. 7,340,077 “Gesture Recognitionspecific hand gestures used as deliminators—to indicatedeliminatorsSystem Using Depth Perceptivebeginning or end of a hand gesture. [p 12, lines 40-45]Sensors”_(Gokturk, 2003)coupling 3D object hotU.S. Pat. No. 8,232,990 “Working with 3Dmethod to modify 3D object on touch screen based on apoints with fingertipsObjects” (Apple, 2010)3D gesture [Claim 7, FIG. 2C]; NB: method requiresfingertips to first start with contact with touch screensurface and then lift off surface to modify 3D object.enriched gestureU.S. Pat. No. 7,665,041 “Architecture forclaims tracking gesture characteristics including at leastlanguage using otherControlling a Computer Using Handone of hand movement, finger orientation, finger count,properties of 3D object Gestures” (Microsoft, 2003finger orientation and hand rotation [Claim 9](velocity, fingertiprotation, etc.)enriched gestureU.S. Pat. No. 8,086,971 “Apparatus, Methodsuses fingertip landmarks, velocity, and rotation of thelanguage using otherand Computer Program Productsfingertips. [page 10, lines 20-30, page 11, lines 45-50]properties of 3D objectProviding Finger-Based and Hand-(velocity, fingertipBased Gesture Command” (Nokia,rotation, etc.)2006)enriched gestureU.S. Pat. No. 7,665,041 “Architecture forflat open hand that can roll in six degrees of motion for alanguage using otherControlling a Computer Using Handrich gesture library [0051, 0054]properties of 3D object Gestures” (Microsoft, 2003(velocity, fingertiprotation, etc.)enriched gestureU.S. Pat. No. 7,340,077 “Gesture Recognitionwhile pushing two fingers on display screen, depthlanguage using otherSystem Using Depth Perceptiveinformation re hand movement used to process twoproperties of 3D object Sensors” (Gokturk, 2003separate commands, although fingers pushed(velocity, fingertipsimultaneously. [p 15, lines 20-30]; can also userotation etc.)fingernails on screen rather than full finger contact todraw a thin line in a virtual canvas. [p 16, lines 30-35]enriched gesture2012/0204133 “Gesture-Based Usercontext specific applications: hand fast down movementlanguage using otherInterface” (Primesense, 2012)interpreted as “mute” button, rather than volumeproperties of 3D objectdecrease on a sound application. [0056](velocity, fingertiprotation etc.)Z0 planeU.S. Pat. No. 7,665,041 “Architecture forhas moveable engagement plane positioned so user canControlling a Computer Using Handcontrol system by moving hand or object through theGestures” (Microsoft 200 3engagement plane and into engagement volume;engagement plane is beginning of engagement volume.[p16, lines 40-50, FIGS. 9A, 9B]. When object is movedthrough engagement frame, feedback is displayed foruser; engagement plane used primarily to start interactioninto engagement volume, and not as focus point forinteraction itself.signal showing 3DU.S. Pat. No. 7,665,041 “Architecture forwhen object moved through engagement frame,interaction zone hasControlling a Computer Using Handfeedback displayed for user [p16, lines 40-50, FIGS. 9A,been entered (buzzer,Gestures” (Microsoft, 20039B].haptic feedback, visualindicator)Smart 3D Context2010/0199228 “Gesture Keyboarding”when entering keyboard characters with gestures, systemAware Applications(Microsoft, 2009is aware that a gesture is used and can attempt character(assumes a 3Dautocorrect due to errors in recognizing gestures [0091]gesture has occurred,e.g., resizing screenand menu options)Smart 3D Context2011/0173574 “”In Application Gesture gestures switch between modes on a computerAware ApplicationsInterpretation” (Microsoft, 2011)application, and can turn on an application, or control(based on the fact thatsome hardware component. [0038, 0041]a 3D gesture hasoccurred, e.g., resizingscreen and menuoptions)
What is needed is a system and method enabling customizable recognition of many types of three-dimensional gestures such that a user can render comfortably and without creating undue stress or strain in the user's arm or hand. Further, there is a need for a system and method that can provide effective user feedback such a graphical user interface (GUI) imagery, perhaps sound, and preferably including haptic feedback, commensurate with a gesture and/or before applying the gesture to the electronic device. Finally such method and system should be low in cost and in power consumption.
The present invention provides methods and systems with such features.